Media device content review and management

ABSTRACT

Systems and methods may be provided to monitor media content on a monitored media device, including multimedia content, to determine whether the content is appropriate for the monitored device or devices, and to take action to remove, filter or otherwise block inappropriate content. The content monitored may include, for example, content such as audio content, video content, images, and text.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/203,453 filed on Nov. 28, 2018, which claims priority to U.S. patentapplication Ser. No. 16/032,027 filed on Jul. 10, 2018, which are bothincorporated herein by reference in their entirety.

TECHNICAL FIELD

The disclosed technology relates generally to content monitoring. Someembodiments relate to content monitoring systems and methods forelectronic devices.

DESCRIPTION OF THE RELATED ART

The advent of smart phone and tablet technology has ushered in an era ofconnectivity and content sharing that fundamentally changed the waypeople conduct their lives. These portable content devices areubiquitous in today's society and they have become a way of life formost people in modern communities. The growth and accessibility ofportable connectivity has arguably also led to an increase in contentsharing and other like applications used in other devices as well, suchas in desktop and laptop computing devices, for example.

Recent years have experienced an explosion of content sharingopportunities among users of electronic devices. Social mediaapplications, gaming applications, video sharing applications, andmessaging applications, are examples of just a few of the many contentsharing opportunities available to users of all ages. The types ofcontent available to users of such devices are seemingly limitless, fromeducational to commercial to entertainment, and everything in between.While the ready availability of vast amounts of varying content canoffer tremendous opportunities for increased productivity, knowledgegrowth and other benefits, if unchecked it can also lead to adverseconsequences. For example, the availability of games and other forms ofentertainment may distract the student from completing his or herhomework assignments; it may also distract an employee from attending tohis or her job duties. As another example, content deemed to beunsuitable for impressionable youth may be accessible to these youth ifunmonitored.

BRIEF SUMMARY OF EMBODIMENTS

According to various embodiments of the disclosed technology, devicesand methods for providing content monitoring for one or more devices tobe monitored. More particularly, in some applications, systems andmethods are provided to monitor media content, including multimediacontent, to determine whether the content is appropriate for the usersassociated with the monitored device or devices, and to take action toremove, filter or otherwise block inappropriate content. The contentmonitored may include, for example, content such as audio content, videocontent, images, and text.

A system for device monitoring may include: an electronic monitoreddevice that may include a communications transceiver a monitoring andanalysis circuit, wherein the monitoring and analysis circuit monitorsdevice media content received, sent or generated by the electronicmonitored device, analyzes the monitored media content and identifiesany monitored media content that exceeds a content tolerance levelestablished for that device as flagged content; generates derived datafor the flagged content; and causes the flagged content and itscorresponding derived data to be transmitted to a server a server, mayinclude a server analysis component; wherein the server stores the mediacontent tolerance level established for the monitored device analyzesderived data using the server analysis component to determine if theflagged content corresponding to the derived data exceeds the mediacontent tolerance level for the monitored device; and generates an alertif it the flagged content may be determined by the server analysiscomponent to exceed the media content tolerance level for the monitoreddevice; and an electronic monitoring device, wherein the electronicmonitoring device receives the alert generated by the server for theflagged content; receives the flagged content associated with the alert;and displays the received flagged content to a user of the electronicmonitoring device.

In one embodiment, the monitoring and analysis circuit may be located inthe monitored device or the monitoring and analysis circuit may belocated in a router connected to the monitored device.

In one embodiment, the monitoring and analysis circuit causes themonitored device to transmit the flagged content to the server and themonitored device encrypts the flagged content prior to transmission andtransmits encrypted flagged content to the server. In anotherembodiment, the monitoring and analysis circuit causes the router totransmit the flagged content to the server and further wherein therouter encrypts the flagged content prior to transmission and transmitsencrypted flagged content to the server. The server may be unable todecrypt the encrypted flagged content sent from the monitored device orthe router to the server. In some embodiments, for example, the servermight not be provided the encryption keys or other data or informationnecessary to decrypt the encrypted flagged content.

The monitoring device may decrypt the encrypted flagged content prior todisplaying the flagged content to the user.

The media content may include at least one of the display output of themonitored device, text input of the monitored device and audio input ofthe monitored device. Content tolerance levels may include a thresholdprobability that the monitored media content contains inappropriatecontent from one or more categories of inappropriate content. Thederived data may include data to determine a probability that thecaptured media content contains inappropriate content. In oneembodiment, the derived data may include text extracted from thecaptured media content. In one embodiment, the derived data may includean identification of third parties associated with the flagged mediacontent.

In one embodiment, monitoring and analysis circuit may employ at leastone of optical character recognition, image recognition, imageclassification, text recognition, voice recognition, speech recognition,and pattern recognition to monitor the media content.

In one embodiment, the server analysis component determines that theflagged media content exceeds a tolerance level if a probabilityindicated by the derived data that the flagged media content containsinappropriate content exceeds a threshold probability set by the mediacontent permission setting that the captured media content containsinappropriate content from one or more categories of inappropriatecontent.

The monitoring device may accept feedback from the user of themonitoring device, the feedback may include an indication from the userwhether or not the flagged content displayed to the user constitutesinappropriate material according to standards of the user. In oneembodiment, the feedback may include at least one of: an indication asto whether the flagged media content should have been determined by theserver analysis component to constitute inappropriate material anindication as to the extent to which the flagged media content containsinappropriate content from one or more categories of inappropriatecontent; a textual input describing characteristics of the capturedmedia content; and a textual input reciting text information containedin the captured media content. The feedback data received by themonitoring device may be used to train at least one of a machinelearning algorithm used by the local analysis component and a machinelearning algorithm used by the monitoring and analysis circuit.

At least one of the server and the monitoring device may remotelyrestrict the functionality of the monitored device, or report or flagthe occurrence of the violation, if the server analysis componentdetermines that the flagged media content violates a content tolerancelevel.

A method for monitoring a device, may include the steps: monitoringmedia content generated by, sent by or sent to an electronic monitoreddevice generating derived data of the monitored media content with amonitored device local analysis component; analyzing the derived datawith a server analysis component to determine if the monitored mediacontent exceeds a content threshold level setting for the monitoreddevice; generating an alert if it the monitored media content may bedetermined to exceed the content threshold level setting; receiving thealert on an electronic monitoring device; and viewing the monitoreddevice media content on the monitoring device.

In one embodiment, the system may further include the monitoring deviceaccepting feedback from the user of the monitoring device, the feedbackmay include an indication from the user whether or not the flaggedcontent displayed to the user constitutes inappropriate materialaccording to standards of the user.

In one embodiment, the feedback comprises at least one of: an indicationas to whether the flagged media content should have been determined bythe server analysis component to constitute inappropriate material anindication as to the extent to which the flagged media content containsinappropriate content from one or more categories of inappropriatecontent; a textual input describing characteristics of the capturedmedia content; and a textual input reciting text information containedin the captured media content.

Other features and aspects of the disclosed technology will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, which illustrate, by way of example, thefeatures in accordance with embodiments of the disclosed technology. Thesummary is not intended to limit the scope of any inventions describedherein, which are defined solely by the claims attached hereto.

BRIEF DESCRIPTION OF THE DRAWINGS

The technology disclosed herein, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The drawings are provided for purposes of illustration only andmerely depict typical or example embodiments of the disclosedtechnology. These drawings are provided to facilitate the reader'sunderstanding of the disclosed technology and shall not be consideredlimiting of the breadth, scope, or applicability thereof. It should benoted that for clarity and ease of illustration these drawings are notnecessarily made to scale.

FIG. 1 illustrates an example of a content monitoring system inaccordance with one embodiment.

FIG. 2 illustrates an example process for content monitoring inaccordance with one embodiment of the systems and methods disclosedherein.

FIG. 3 illustrates another example process for content monitoring inaccordance with one embodiment of the systems and methods disclosedherein.

FIG. 4 illustrates an example data flow of the example embodimentillustrated in FIG. 3.

FIG. 5 is a diagram illustrating an example implementation of amonitored device in accordance with one embodiment.

FIG. 6 is a diagram illustrating an example of content capture analysisfor a monitored device in accordance with one embodiment.

FIG. 7 is a diagram illustrating an example implementation of amonitoring device in accordance with one embodiment.

FIG. 8 is a diagram illustrating an example of captured content reviewand feedback by a monitoring device in accordance with one embodiment.

FIG. 9 is an example illustrating a message string that can be monitoredat a monitored device for potential inappropriateness.

FIG. 10 is a diagram illustrating an example of review and feedback on amonitoring device in accordance with one embodiment.

FIG. 11 illustrates another example of review and feedback one amonitoring device in accordance with one embodiment.

FIG. 12 is a diagram illustrating an example of a false positive inaccordance with one embodiment.

FIG. 13 illustrates yet another example of supervisor feedback inaccordance with one embodiment.

FIG. 14 illustrates an example computing module that may be used inimplementing various features of embodiments of the disclosedtechnology.

The figures are not intended to be exhaustive or to limit the inventionto the precise form disclosed. It should be understood that theinvention can be practiced with modification and alteration, and thatthe disclosed technology be limited only by the claims and theequivalents thereof.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the technology disclosed herein are directed towarddevices and methods for providing content monitoring for one or moredevices to be monitored. More particularly, in some applications,systems and methods are provided to monitor media content, includingmultimedia content, to determine whether the content is appropriate forthe monitored device or devices, and to take action to remove, filter orotherwise block inappropriate content. The content monitored mayinclude, for example, content such as audio content, video content,images, and text.

For example, in some implementations content generated by, received by,or otherwise present on a monitored device can be reviewed to determinewhether the content is appropriate according to a definedappropriateness standard for that device or its user. The content can betested, and if potentially inappropriate content is detected on themonitored device, information pertinent to the potentially inappropriatecontent may be sent to a server for further evaluation. This informationmay include, for example, the potentially inappropriate content itselfor portions thereof, and derived data corresponding to the content andits potential inappropriateness. The server may analyze the informationto determine whether the content is inappropriate. If so, the server maygenerate an alert to a monitoring device. The monitoring device may begiven permission to review the potentially inappropriate content thatcaused the alert and make a final determination as to whether or not thepotentially inappropriate content is indeed inappropriate. If themonitoring device determines that the content is indeed inappropriate,actions can be taken to restrict the monitored device from receivingthis content. For example, the content can be removed from the monitoreddevice, or it can be blocked or disabled such that the user of themonitored device cannot access or view the content. As another example,the source of the inappropriate content can be identified (e.g., by anIP address, email address, SMS or MMS address, and so on) and thatsender blocked from sending further content to the monitored device.

In various embodiments, the monitoring device may also generate feedbackabout the content which can be used to update the analysis mechanismsused by the server and the local analysis mechanisms used by themonitored device to evaluate content. For example, a user of themonitoring device may determine that the flagged content is not reallyinappropriate according to his or her standards, or that it was flaggedin error, and this feedback can be used to adjust the algorithms used toflag content as potentially inappropriate. In various implementations,AI and machine learning techniques can be used to update the analysismechanisms used by the various devices to refine the decision-makingprocess for flagging content as potentially inappropriate according tothe reviewer's standards for inappropriateness.

Before describing further embodiments in greater detail, it is useful toprovide a high-level description of embodiments of the systems andmethods disclosed herein. FIG. 1 illustrates an example of a contentmonitoring system in accordance with one embodiment. More particularly,FIG. 1 illustrates an example of a monitoring device 102 that ismonitoring a monitored device 104 through the use of a monitoring server106. Devices monitored can include, for example, smart phones, tablets,media players, computers (desktops, laptops, etc.), automobile headunits, workstations, and other content devices. Likewise, monitoringdevices 102 may include, for example, smart phones, tablets, mediaplayers, computers (desktops, laptops, etc.), workstations and otherprocessing devices. Server 106 may include a computing platformconfigured to provide monitoring of content and sharing of contentbetween monitoring devices 102 and monitored devices 104.

As also illustrated in FIG. 1, monitoring devices 102, monitored devices104 and servers 106 may communicate with one another using one or morecommunication networks 108. For example, the devices may connect via oneor more communication platforms such as Bluetooth, IEEE 802.11 (e.g.,WiFi), Ethernet, Internet, cellular data networks (e.g., 3G, 4G, 5G,etc.), or other communications networks. As described in exampleembodiments presented below, information exchanged via communicationnetworks 108 may include monitored content, content flagged aspotentially inappropriate, data relating to the content, and othercommunications between or among monitoring devices 102, monitoreddevices 104 and servers 106.

Although not illustrated in FIG. 1, in some applications a monitoringdevice 102 and a monitored device 104 can be the same device. Forexample, in one-device households, the single device may be sharedamongst various family members. These family members sharing the devicemight include parents and children who each share the household device.Device sharing such as this is not limited to 1-device households, buteven households with multiple devices may share one or more of mediadevices between a parent and a child, or among multiple parents andchildren. These scenarios are not limited to a household, but can alsoapply in other environments where one or more devices are shared bysupervisors and supervisees.

Accordingly, embodiments can be implemented in which supervisorfunctions such as those described herein can be implemented on the samedevice as the supervisee functions. In such embodiments, contentmonitoring, feedback and control can be managed using, for example, userIDs, login information, or other user identification information.Likewise, access to control and feedback functions can be limited tosupervisors or other authorized users using similar useridentification/authentication information.

Before describing embodiments in more detail, it is useful to describe afew example applications for the systems and methods disclosed herein.For instance, in one scenario the system may be configured such that aparent or guardian may use a monitoring device (e.g., monitoring device102) to monitor and control content (e.g., text, images, videos or othercontent) on their child's or children's device or devices (e.g.monitored device 104). In another application, an employer may use amonitoring device to monitor and control content on the devices of itsemployees. As yet another example, a teacher or school administrator mayuse a monitoring device to monitor and control content on the devices ofstudents. For ease of discussion, parents, bosses, employers, teachers,administrators, or other persons using a monitoring device or devices tomonitor another device may be referred to herein as a supervisor.Similarly, children, workers or employees, contractors, students orother persons subject to content monitoring on a monitored device ordevices may be referred to herein as a supervisee. This terminology isnot intended to imply or require a boss/employee relationship, but ismerely adopted to provide an efficient way to describe persons who aremonitoring the content accessed or used by another, and those personswhose content access or other use is being monitored.

FIG. 2 illustrates an example process for content monitoring inaccordance with one embodiment of the systems and methods disclosedherein. This process is described in terms of an example scenario inwhich a supervisor (e.g., parent, employer, teacher, etc.) uses acontent monitoring system to monitor content on the device of asupervisee (e.g., child, employee, student, etc.). Referring now to FIG.2, at operation 240, the supervisor defines one or more thresholds forinappropriateness of content on the supervisee's device. For example,the supervisor can define thresholds for content characteristics suchas, for example, violence, nudity, illicit content, profanity,gaming/entertainment, other subject matter, or other content criteria.Additional examples of inappropriateness thresholds are provided below.Accordingly, whether content is inappropriate or not can be as definedby one or more supervisors for one or more supervisees. As discussedbelow, this level can be adjusted or refined using feedback gatheredfrom the one or more supervisors. In other embodiments, whether contentis inappropriate can be predefined, and this predefined threshold can befurther refined or adjusted based on feedback or based on user (e.g.,supervisor) preferences.

At operation 242, the selected tolerance thresholds are set on thesupervisee's monitored device. In some implementations, monitoreddevices can have more than one tolerance threshold or set of tolerancethresholds. For example, a monitored device that supports multiple usersmay have different tolerance thresholds set for the different usersupported by that device. These thresholds could be stored on the deviceand implemented operationally based on user login, for example. Asanother example, a monitored device may have different thresholds for auser that very based on external factors such as, for example, locationof the device, day and time of day (e.g., during school or afterschool),and so on. As yet another example, a monitored device may have differenttolerance levels that can be activated remotely by a monitoring deviceor locally via a password-protected interface. In some embodiments, thetolerance levels can be stored on the monitored device for localanalysis of content at the monitored device. The media content tolerancesettings can define the relationship between the monitored device andthe monitoring device that establish those tolerance levels for themonitored device.

At operation 244, the monitored device captures media content, analyzesthe captured content for comparison against the tolerance thresholds andgenerates data about the captured content. The content monitored andanalyzed can include, for example, audio data, text data, image data(still or video), input from a camera, speech, or other content. Themonitored device sends the content and the captured data to the contentmonitoring server for further analysis. In some embodiments, the contentcan be encrypted before sending it to the server.

At operation 246, the server analyzes the data and determines whetherthe content is likely inappropriate based on the thresholds establishedfor the monitored device. In embodiments where the content itself isencrypted such that the server cannot access the content, the serverdetermines the probability that the content violates a tolerancethreshold based on the derived data for that content. If the contentappears to violate a threshold based on this analysis, the server cangenerate an alert and send that alert to the monitoring deviceindicating the discovery of potentially inappropriate content on themonitored device. The alert may include, for example, a digitalnotification, text message, email, or other message or communication totrigger an alert or notification on the monitoring device. In variousembodiments, the server can store the content and its correspondingderived data, and can further provide the derived data and the contentto the monitoring device. The server may be implemented as a cloud-basedserver or distributed server, or can be a server at a particularfacility.

At operation 248, the monitoring device receives the alert. Thesupervising device may retrieve the associated content from the serverand the supervisor may review the content him or herself to determinewhether it is inappropriate based on the standards of the supervisor.The supervisor may then respond to the server with feedback indicatingwhether the supervisor has determined the content to indeed beinappropriate.

At operation 250, appropriate action can be taken vis-à-vis any contentdetermined to be inappropriate. For example, one or more actions can betaken such as, for example, deleting the inappropriate content, blockingcommunication with the source/destination of the inappropriate content,disabling the application or other instrumentality running theinappropriate content (e.g., MMS service, video player, audio player,etc.), disabling the monitored device itself, and so on. In someembodiments, these action can be taken automatically, or user input maybe required before any action is initiated in response to theinappropriate content. For example, a settings menu can include theappropriate options to allow the supervisor to choose whether to enableautomatic action were content is determined to be inappropriate or torequire supervisor input before action is taken. A menu may also allow asupervisor to elect the type of action taken. Actions may include one ormore of, for example, removing inappropriate content from the device,blocking senders or recipients of the inappropriate content fromcommunicating with the device, blocking the content itself, shuttingdown or disabling the device, shutting down or disabling the applicationon which the offending content occurs, shutting down or disablingcommunication with the device, warning the supervisee about theoffensive content, and other actions as may be appropriate to addressthe supervisee's handling of inappropriate content.

FIG. 3 illustrates another example process for content monitoring inaccordance with one embodiment of the systems and methods disclosedherein. FIG. 4 illustrates an example data flow of the exampleembodiment illustrated in FIG. 3. This example is described in terms ofan implementation in which a server 440 is used to facilitate contentmonitoring of a monitored device 430 by a monitoring device 450.

Referring now to FIGS. 3 and 4, at operation 330, media content onmonitored device 430 is captured. The content can be captured as it isentering monitored device 430. For example, in the case of a device withcommunications capability such as a smart phone, tablet or other likedevice, incoming messages such as emails, SMS messages, MMS messages, IMmessages, can be monitored. As another example, downloaded or streamingcontent such as content received through an Internet, ethernet, USB,cellular data, Wi-Fi, Bluetooth or other connection can also bemonitored. As yet a further example, text being entered by a user (e.g.,via keyboard, speech recognition, or other input mechanism), picturesuploaded by user or other content introduced to the device can bemonitored. In addition to or instead of capturing content as it entersmonitored device 430, content stored in memory of monitored device for30 can also be captured for analysis.

At operation 332, the captured content is analyzed with a local analysiscomponent to determine whether any of the monitored content includespotentially inappropriate content. In some implementations, this contentmonitoring and analysis can be performed by a monitoring and analysiscircuit included on monitored device 430. The local analysis caninclude, for example, analyzing captured media content using tools suchas text recognition tools, OCR tools, image and pattern recognitiontools, image classification tools, voice and speech recognition tools,and so on. The text, image, speech and other forms of content can bereviewed to determine whether the content violates the tolerance levelsestablished for the monitored device. For example, the presence ofprofane or illicit language in text messages, email messages or speechmay cause the content to be flagged as inappropriate. As anotherexample, pattern or image recognition may be used to identify sexualcontent, nudity or violence.

The monitoring and analysis circuit can be configured or programmed withvarious threshold settings for flagging content as potentiallyinappropriate. For example, text comparison recognition, textual andcontextual analysis, image and pattern recognition, and other analysistools can be used to determine content that may potentially beinappropriate. Data regarding the potentially inappropriate content canbe derived from the content to aid in further analysis. The derived datacorresponding to the content can include, for example, data pertainingto the type of content flagged, specifics of the content flagged,characteristics of the content flagged, categories that the content mayviolate, tolerance levels for the monitored device, addresses to/fromwhich the content was sent, and so on.

At operation 334, the potentially inappropriate content, or portionsthereof, and the derived data can be transmitted to the server 440 forfurther analysis. In some embodiments, the content and the derived datacan be encrypted for secure transmission. Server 440, and monitoringdevice 450 can be provided the appropriate decryption algorithms anddecryption keys to decrypt some or all of the content and data forfurther analysis. In some embodiments, the content itself is encryptedusing encryption techniques or encryption keys such that server 440cannot decrypt the content itself, but can only decrypt other dataassociated with the content for analysis. Using different levels orforms of encryption can be implemented to protect the privacy of varioususers of the system by not exposing the content to a third-party server.In this way, server 440 can analyze data to determine an appropriatenesslevel of the content without being permitted access to the contentitself.

As illustrated in FIG. 4, server 440 can store the data and the contentfor further analysis. At operation 336, server 440 analyzes the deriveddata with a server analysis component to determine whether the contentis in fact inappropriate. As noted above, in some implementations thecontent itself may be secured (e.g., encrypted) such that the server 440cannot access the content itself, but relies only on derived data. Forexample, server 440 might not be given the decryption keys or otherinformation or materials necessary to access the security content. Insome embodiments, the system can be implemented such that the server 440can access limited content snippets to make the determination. Securingthe content itself from review by server 440, and allowing server 440 tomake a determination based on derived data alone (or limited contentsnippets) can help to preserve the privacy of the users and can alsoshield server 440 from information it should not have or may not want.

If server 440 determines that the content is not inappropriate, themonitoring continues and an alert need not be sent to monitoring device450. However, in some implementations, server 440 can be configured tosend alerts or updates on a regular basis even if no inappropriate orpotentially inappropriate content is detected. Server 440 can also beconfigured to send regular reports with information such as, forexample, the amount and types of content evaluated (e.g., based on thederived data), the results of the analysis and other summary orreporting information. Content received at the server can be stored forpredetermined periods of time such that the supervisor can go back andreview stored content.

If, on the other hand, server 440 determines that the content isinappropriate (operation 338), server 440 generates an alert and sendsthe alert to monitoring device 450. This is illustrated at operation340. Server 440 can send the associated content to monitoring device450, or monitoring device 450 can retrieve the data from server 440. Thecontent can be decrypted if necessary and reviewed by the supervisor.Where monitoring device 450 and monitored device 430 are the samedevice, the inappropriate content need not be delivered to themonitoring device 450 as it is already there. In such implementations,server 440 can simply send an identification to monitoring device 450identifying the content items to be reviewed.

The monitoring operation can continue while a supervisor at monitoringdevice 450 reviews the relevant content. As noted above, tolerancelevels for the monitored device can be sent to server 440 as part of thederived data so that server 440 can conduct the analysis based on thedevice tolerance settings. Tolerance levels can also be stored at theserver for the monitored devices.

Monitoring device 450 can decrypt the content if necessary so that thesupervisor can view the content to see what the supervisee is viewing orcreating on monitored device 430. At operation 342, the supervisor cangenerate feedback regarding the content and its appropriateness and sendits feedback to server 440 or monitored device 430 or both for updatingthe algorithms used to analyze the content. For example, the supervisormay indicate through the feedback that the flagged content is not reallyinappropriate. Machine learning and neural network techniques can beused for the analysis process and this feedback can be used by themachine learning algorithms to refine the analysis (e.g., to furthertrain the neural network) or update the threshold levels.

As illustrated by flowline 344, the operation continues with content onthe monitored device being monitored and analyzed for potentialinappropriateness. In various implementations, monitoring and analysiscan be continuing and ongoing in real time or near real time as thedevice is operating and receiving content from external sources or ascontent is generated by user. In further implementations, the monitoringanalysis can be performed on stored content that was collected duringoperation, such as in non-real-time. In some implementations, monitoringcan be enabled and disabled by the user (e.g., by the supervisor)manually so that monitoring can be switched on and off on demand.

In some embodiments, the system can be configured to monitor everycommunication and every item of content received by or entered into thedevice. In other embodiments, the system can be configured such thatonly selected types of content or content from selected sources aremonitored. For example, the system can be configured to monitor onlyemail correspondence; or to monitor only email, SMS and MMScorrespondence; or to monitor only video and still image content; and soon. This selective monitoring by content type (e.g., type of contentitem or source of content items) can be established by the system, or bythe user. For example, a supervisor setting up monitoring for amonitored device may be given the opportunity to designate the types ofcontent monitored for that device. As a further example, a parentsetting up monitoring for his or her child's tablet may desire tomonitor all types of content from all sources, whereas an employersetting up monitoring for a senior-level employee's computer mightchoose to monitor only selected types of content or selected contentsources.

Implementations can also be configured in which periodic monitoring orspot checking of content can be made, as opposed to continuous,full-time monitoring. For example, the monitoring can be set up so thatthe specified content items and sources are monitored and analyzed atdetermined intervals of time or for determined levels of data. Forexample, the system can be configured to check various content atdifferent intervals. As a further example, the system can be configuredto check one in 10 or 20 emails, one out of every 5 or 10 videos, 1 outof every 5 text messages, and so on. As another example, the system canbe configured to check various content items at given time intervals.The monitoring periodicity does not have to be uniform as the aboveexamples suggest. Monitoring intervals or times can also be set asrandom times or nonuniform monitoring intervals.

As these examples illustrate, different levels of periodic or spot checkmonitoring can be performed such that the system is not required tomonitor each and every communication or content item that crosses themonitored device. In some embodiments, the level of monitoring can beuser controlled such that the supervisor, for example, can establish thefrequency of monitoring of the content items. In further embodiments,the user (e.g., supervisor) can choose different monitoring frequenciesfor different content items or different communication channels on thedevice. As noted above, embodiments can also be implemented in which theuser can turn monitoring on and off manually (e.g., for on-demandmonitoring).

In further embodiments, the system can be implemented such that ifinappropriate content is detected (e.g., through analysis by themonitored device, analysis by the server, or analysis and feedback fromthe monitoring device) the level of monitoring is increasedautomatically. In some implementations, the level of monitoring can beincreased across the board for all content types (e.g. types of contentand content sources) being monitored. In other implementations, thelevel of monitoring can be increased for the content sources or contenttypes corresponding to the inappropriate content identified. Likewise,the system can be configured such that if a determined amount of timelapses or volume of content is monitored without detecting inappropriatecontent, the monitoring frequency can be decreased. Again, this can beconfigured to occur automatically or based on user input.

FIG. 5 is a diagram illustrating an example implementation of amonitored device in accordance with one embodiment. In the exampleillustrated in FIG. 5, monitored device 530 includes communicationstransceiver 502, processor 504, memory 506, device storage 508, userinterface 510, monitoring and analysis circuit 512, content data storagedevice 514, and power supply 516.

One or more communications transceivers 502 can be included withmonitored device 530 for wired or wireless communications interfacessuch as, for example, Wi-Fi, Bluetooth, ethernet, USB, cellular,cellular data, and other communications interfaces. In variousapplications, it is anticipated that communications transceivers 502would include one or more transmitters and receivers to providebidirectional communications between monitored device 530 and otherdevices. In other applications, monitored device 530 can include just atransmitter or just a receiver for one-way communications.Communications transceivers 502 can transmit and receive content that ismonitored on a monitored device 530. Communications transceivers 502 canalso transmit and receive other content and other data and information.Although not illustrated in the example of FIG. 5, the monitored devicemay also include one or more antennas for wireless communications orports/connectors for wired communications.

Processor 504 and memory 506 can be provided to perform traditionalfunctions of monitored device 530. For example, processor 504 can beincluded to control the operations of monitored device 530 and toexecute applications and other instructions on monitored device 530.Processor 504 can be implemented as one or more single- or multi-coreprocessors, CPUs, DSPs, or other processor units, and may be configuredto execute instructions to execute applications and otherwise carry outthe functions of monitored device 530. For example, messagingapplications or instructions therefore can be stored in memory 506 andused by processor 504 to conduct messaging operations.

Memory 506, can be made up of one or more modules of one or moredifferent types of memory, and in the illustrated example is configuredto store data and other information as well as operational instructionsthat may be used by the processor 504 to operate monitored device 530.Likewise, device storage 508 can be made up of one or more modules ofone or more different types of memory. Device storage 508 may be used,for example, to store content, applications, or other data andinformation in conjunction with the operation of monitored device 530.For example, device storage 508 may include built-in memory or a memorycard slot (e.g., microSD) to store apps, photographs, videos, or othermaterials on monitored device 530.

Although not illustrated, monitored device 530 may also include otherfunctional components such as a camera, for example. Still or videoimages captured by the camera can be included in the content monitoredby monitoring and analysis circuit 512.

User interface 510 can provide an I/O interface for user interactionwith monitored device 530. In the illustrated example, user interface510 includes a display to display text, graphics, video, or otherinformation to the user, a keyboard to accept user input, and a speakerfor audio output. In some embodiments, user interface 510 can be atouchscreen user interface that provides a display and a touchscreenkeyboard interface. Buttons, knobs, dials, or other user inputs can alsobe included. The speaker can be implemented to include one or more audiospeakers or other audio emitters to provide audio content to the user.Although not illustrated, a microphone or other speech transducer can beincluded to accept audio input from the user.

Monitoring and analysis circuit 512 can be included to providemonitoring and analysis of various content items to determine theappropriateness of the content in accordance with one or more contentlevel settings. Monitoring and analysis circuit 512 can be implementedusing a processor and memory (e.g., like processor 504 and memory 506),as dedicated circuitry (e.g. ASIC, FPGA or other circuitry), or acombination of the foregoing. In some embodiments, the functions ofmonitoring and analysis circuit 512 can be performed using sharedresources of monitored device 530, or monitoring and analysis circuit512 can be dedicated resources for the monitoring and analysisfunctions. For example, the functions of monitoring and analysis circuit512 can be performed by an application executing on processor 504,whereas in other embodiments, monitoring and analysis circuit 512 can beseparate from processor 504.

In some implementations, monitoring and analysis circuit 512 can includea neural network operating on one or more processors and trained todetect various forms of inappropriate content. For example, a neuralnetwork can be initially trained using a variety of content itemsalready identified as appropriate or inappropriate in the trained netinstalled on the device. The training set of content items can be fed tothe neural network with the results so that the neural network cantrained with an initial data set. A trained model can be used for agiven application, or it can be propagated across multiple applications.Similarly, different models can be trained and these different modelspropagated to different devices or different classes of devices. Forexample, models can be trained with different tolerance levels or tolook for different classes of inappropriate content.

When feedback is provided (e.g. by a supervisor), this feedback can beused to further refine the training of the neural network. For example,the neural network may flag as inappropriate images similar to thosewith which it was trained. Where the supervisor feedback indicates thatthese flagged images are inappropriate, that can reinforce the training.Where the supervisor indicates that the flagged images are notinappropriate, the neural network can be refined to discriminate thoseimages. This process of updating and refining the training can continuefor a period of time or throughout the life of the device and canfurther define what constitutes inappropriate content in accordance withthe user's (e.g. the system trainer's or supervisor's) threshold.

The system can be configured such that the output of the neural networkis a confidence score. For example, the neural network may be configuredto conclude or to output a result indicating that particular content isinappropriate with an X % confidence level. Where the confidence levelexceeds a predetermined threshold percentage (e.g. a threshold set basedon the supervisor's level of tolerance for content), this can indicatethat the content is inappropriate. Where the confidence level does notexceed that threshold, this can indicate that the content is notinappropriate.

In some embodiments, monitoring and analysis can also be performed on adevice separate from the monitored device. For example, some or all ofthe functions of monitoring and analysis circuit 512 can be performed ona router with which the device is connected. For example, tools on therouter that enable viewing and analysis of HTTP, HTTPS, HTTP/2 andenabled TCP port traffic accessed from, to, or via the monitored devicecan be used to perform these functions whether at the device itself orat the router.

Content data storage device 514 can be provided to store one or more ofcontent items flagged by monitoring and analysis circuit 512 aspotentially inappropriate, derived data associated with the flagcontent, and device tolerance levels for the monitored device. Contentstorage device 514 can occupy an allocated or shared portion of memory506 or device storage 508, or it can be a storage device separatelyprovided for this content storage. Device tolerance levels stored incontent data storage device 514 can include, for example, the monitoringlevel or scrutiny level established for the monitored device 530.

An encryption circuit can also be included to encrypt content and derivedata transmitted to the server for further analysis. As noted above,different levels or types of encryption or different encryption keys canbe used for different content items to provide different levels ofaccess for access by the server and the monitoring device. In someembodiments, the encryption can be performed by monitoring and analysiscircuit 512, processor 504, communications transceiver 502 or othercircuitry in monitored device 530.

Power supply 516 can include any of a number of power sources used toprovide electrical power to monitored device 530 and its variouscomponents. Power supply 516 can be implemented, for example, as abattery (rechargeable or otherwise), a capacitor or capacitive storagedevice, an AC-to-DC or DC-to-DC power converter, photovoltaic cells, ora combination of one or more of the foregoing.

FIG. 6 is a diagram illustrating an example of content capture analysisfor a monitored device in accordance with one embodiment. In thisexample, monitored device 630 is a handheld device, although monitoreddevice 630 can be any of a number of electronic content devices. In thisexample, the types of media content captured include audio, stillimages, speech or text, and video images. Local analysis block 644includes circuits to analyze the captured media content includingcircuits for one or more functions such, as for example, textrecognition, optical character recognition, image recognition, patternrecognition, and speech recognition. In terms of the example illustratedin FIG. 5, these can be part of monitoring and analysis circuit 512, forexample. The device can also include circuitry to analyze a recognizedcontent items to determine whether the content is potentiallyinappropriate.

FIG. 7 is a diagram illustrating an example implementation of amonitoring device in accordance with one embodiment. In the exampleillustrated in FIG. 7, monitoring device 730 includes communicationtransceiver 702, processor 704, memory 706, device storage 708, userinterface 710, content review and feedback circuit 712, content storage714, and power supply 716.

With the exception of content review and feedback circuit 712, theillustrated components of monitoring device 730 in this example of FIG.7 can be implemented similar to the like numbered components ofmonitored device 530 in FIG. 5.

Content review and feedback circuit 712 can be included to providecontent to the supervisor for monitoring device 730 so that thesupervisor can review the content and determine whether the content isappropriate in accordance with his or her views of the desired level ofappropriateness of the content. Accordingly, content review and feedbackcircuit 712 can provide the content to the supervisor via a userinterface 710. Content review and feedback circuit 712 can beimplemented using a processor and memory (e.g., like processor 704 andmemory 706), as dedicated circuitry (e.g. ASIC, FPGA or othercircuitry), or a combination of the foregoing. In some embodiments, thefunctions of content review and feedback circuit 712 can be performedusing shared resources of monitoring device 730, or content review andfeedback circuit 712 can be dedicated resources for the monitoring andanalysis functions. For example, the functions of content review andfeedback circuit 712 can be performed by an application executing onprocessor 704, whereas in other embodiments, content review and feedbackcircuit 712 can be separate from processor 704.

Content review and feedback circuit 712 can also accept input from thesupervisor regarding the supervisor's opinion of the appropriateness ofthe flag content. For example, the supervisor can enter input agreeingthat the content is inappropriate, or disagreeing and instead indicatingthat the content is appropriate or okay for the supervisee. In variousembodiments, the system can also accept reasons why the content isappropriate or inappropriate such that the machine learning algorithmsused to identify and flag potentially inappropriate content at themonitored device can be updated in accordance with the supervisor'sopinions of appropriateness. The supervisors input can be made usinguser interface 710 such as, for example, by touch screen or keypadinput, by speech input, or otherwise. Menus or other display interfacescan be provided to the supervisor to facilitate review and feedback.Generated feedback can be sent to the server and the monitored devicesuch as by communication transceiver 702.

Content review and feedback circuit 712 can also retrieve the flaggedcontent that is stored on the server so that the flag content can bedisplayed or otherwise presented to supervisor for review via the userinterface 710. Although not illustrated, appropriate encryption anddecryption algorithms can be used to safeguard the transfer of contentand feedback.

FIG. 8 is a diagram illustrating an example of captured content reviewand feedback by a monitoring device in accordance with one embodiment.In this example, monitoring device 830 is a handheld device, althoughmonitoring device 830 can be any of a number of electronic contentdevices. In this example, content review 822 (e.g., by content reviewand feedback circuit 712) can present the content of the supervisor sothe supervisor can view the flagged media content for review andcomment. Content review 822 can also allow the supervisor to requestmedia content for review. In some embodiments, the server can send analert to the monitoring device that content is available for monitoring.This, in turn, can trigger a notification to the supervisor that thereis content to be reviewed. It is noted that the supervisor need not waitto receive a notification, but can also initiate review by checkingstatus of alerts received by the monitoring device from the server, orby accessing the server from the monitoring device to determine whetherthere are additional content items ready to review.

The initiate action block 824 (e.g., by content review and feedbackcircuit 712) can allow supervisor, or the device itself, to request morecontent be gathered for the particular analysis, or to increase theamount of content gathering on a go forward basis. The initiate actionblock 824 can also be used to enforce restrictions on the monitoreddevice. This can include, for example, shutting down or locking themonitored device, disabling particular applications of the monitoreddevice, blocking content from entering the monitored device, shuttingdown particular communication channels of the monitored device. Therestrictions may also include slowing down or throttling aspects of themonitored device, such as, for example, reducing the processing speed orabilities of the monitored device, slowing down the data throughput ofthe monitored device, and so on. The restrictions placed on themonitored device may vary based on the tolerance settings of the deviceor based on the level of inappropriateness of the received content. Forexample, more serious infractions may lead to more serious consequenceson the monitored device.

In some embodiments, the restrictions enforced on the monitored devicecan be imposed in response to action taken by the supervisor. Forexample, after reviewing the content the supervisor may determine thatit is indeed inappropriate and request that the restrictions be put intoplace. In other embodiments, the restrictions can be imposedautomatically upon the generation of an alert by the server. In suchembodiments, the supervisor may lift the restrictions such as, forexample, after the supervisor has review the content and determined itslevel of inappropriateness.

The submit feedback block 826 (e.g., by content review and feedbackcircuit 712) can be used to allow the supervisor to submit feedbackregarding the content reviewed by the supervisor. This can include, forexample, binary feedback (e.g. the flagged content is appropriate orinappropriate), quantitative feedback such as the level ofinappropriateness of the flagged content reviewed (e.g., on a scale of 1to 5 or 1 to 10, a rating scale, or other quantitative review), orqualitative feedback (e.g., how the supervisor feels about theappropriateness of the content on a qualitative or descriptive basis, ortext input describing the captured content). Some examples ofqualitative and quantitative feedback are described below.

FIG. 9 is an example illustrating a message string that can be monitoredat a monitored device for potential inappropriateness. In this example,a supervisee is using monitored device 932 engage in an SMS conversationwith another party. In this example exchange, the supervisee and theother party to the SMS conversation our students in middle school. Inthis example, the other person asks the supervisee whether she wants tosmoke after school. The supervisee responds that she would rather skipclass instead, to which the other person replies that they cannot skipclass because they have a test.

In various embodiments, text recognition algorithms can be run on thewords and phrases in the SMS message string to determine whether anyflags are raised indicating potentially inappropriate activity. Forexample, words like “smoke” or phrases like “skip class” may be set toraise flags for inappropriate content. As another example, the[expletive] may trigger a tolerance level setting relating to profanity.In other examples, vulgar language, sexually explicit language,threatening or bullying language, language pertaining to inappropriateactivities, to name a few, may also be set to raise flags forinappropriate content. Further to this example, where the system is setsuch that the word “smoke” or the phrase “skip class” is set to raiseflags for inappropriate content, the content may be flagged aspotentially inappropriate, data can be derived from the content to allowfurther analysis, and the content and derived data sent to the server(or in other embodiments directly to the supervisor or supervisors) forfurther analysis and review. As noted, the content can be encrypted suchthat the server cannot access the actual content itself, but the servercan still access the derived data to perform further analysis. Thecontent can include a transcript of the SMS message, or screenshots ofthe SMS message.

The derived data may include, for example, just the inappropriate wordsthat were flagged (e.g., without revealing the entire conversation),date and time information during which the conversation took place, thecontext in which the inappropriate words arose (e.g., an SMS message), acontact in the messaging application, or other information that would beuseful to allow the analysis by the server. This data may also includemetadata regarding the analysis and judgments made by the analysis toolas well as weights used in performing the analysis (e.g., for a neuralnetwork). As another example, the data can include information such ascategory of the content, class of the content, and a confidence level inthe result of the analysis that took place on the device.

As another example, image recognition and analysis tools may be used toflag inappropriate images such as, for example, nudity, obscenegestures, violence, and so on. In this example, derived data may includedata such as, for example, date and time the image was received, andidentification of the sender or source of the image, and imageclassification (e.g., nudity, violence, etc.) or other derived data toallow the server to make a determination without having to access theactual content itself.

FIG. 10 is a diagram illustrating an example of review and feedback on amonitoring device in accordance with one embodiment. In this example, asupervisor reviews flagged captured content 1022 on the screen of his orher monitoring device 1030. Also displayed on the device screen is abinary feedback option that allows the supervisor to respond in a yes orno fashion whether the content that he or she reviewed is actuallyinappropriate content. In this example, the choices are an X to indicate“no” or a checkbox to indicate that “yes” the content is inappropriate.In other implementations, the words “YES” and “NO” can be used, or othersymbols and descriptors can be used to present feedback choices to thesupervisor. This example also illustrates an input allowing thesupervisor to select whether he or she wishes to enforce a devicerestriction. This example illustrates the selection with an arrow button(similar to a “play” button), however other designations can be used.

FIG. 11 illustrates another example of review and feedback on amonitoring device in accordance with one embodiment. Similar to theprior example, in this example a supervisor reviews flagged capturedcontent 1122 on the screen of his or her monitoring device 1130.However, in this example, the user interface allows the supervisor toenter quantitative feedback regarding the levels of inappropriateness ofthe reviewed content. More particularly, this example illustrates a userinterface that allows the supervisor to enter quantitative feedback forthree categories of potentially inappropriate content: violence, sexualcontent, and language. Accordingly, the supervisor can rate the contentdifferently in each of the three categories and this feedback can beused to refine the analysis algorithms based on the supervisorstolerance for material in these three categories. A machine learningalgorithm can accept input from the supervisor over a number ofdifferent content items and assimilate that information to refine thesettings for the monitored device. For example, feedback indicating thata particular content item was not inappropriate may be used to refinethe algorithm used to assess the content. The algorithm may learn overtime that particular images are not inappropriate. In someimplementations, this can be used to update the algorithm for thatsupervisor or installation alone, whereas in other implementations thisinformation can be used to refine the algorithm for multiple differentusers across a plurality of offices, households, etc.

FIG. 12 is a diagram illustrating an example of a false positive inaccordance with one embodiment. In this example, similar to the exampleof FIG. 10, the word “smoke” is included in the SMS message string. Alsosimilar to the example of FIG. 10, the presence of this word may triggera flag for this content, causing the content to be sent to the serverfor analysis and ultimately to the supervisor for review. Howeverbecause the context is not smoking cigarettes after school, but isinstead referring to a permitted bonfire, the supervisor may flag thecontent as okay, or not inappropriate. Accordingly, the supervisor mayin this instant select the “X” selection on the touch sensitive displayscreen. Although the system may continue to alert on the term “smoke” itmay learn through this feedback that when discussed in the context of abonfire, for example, the term is not okay. As a further example, if theterm “smoke” was flagged over numerous occasions, and each time thesupervisor's feedback is that this is not inappropriate, the system canupdate the settings to no longer generate an alert based on the presenceof this term. For example, a supervisor may not care whether thesupervisee takes a smoke break.

FIG. 13 illustrates yet another example of supervisor feedback inaccordance with one embodiment. In this example, supervisor operatingmonitoring device 1330 wishes to add text that was not flagged or wishesto add flag text to further reinforce its importance in theappropriateness determination. Accordingly, the supervisor can select onthe user interface to input text that it wishes to either reinforce orflag. In this example, the supervisor may enter “let's skip class” inthe input box 1334.

In the examples illustrated above with reference to FIGS. 1-4, and inother embodiments discussed herein, a server may be included to analyzeat least the derived data, and in some instances the content, todetermine the appropriateness of the flagged content. As noted above, insome embodiments the content can be encrypted such that the servercannot access the content itself, but can only access the derived datafor its analysis. In such embodiments, the system may still beimplemented such that the server can store the encrypted content foraccess by the supervisor or supervisors on a monitoring device so thatthe supervisor could review the content for appropriateness.

In other embodiments, the system can be implemented without a server. Insuch implementations, the monitored device can monitor and analyzecontent and send captured content to the monitoring device withoutinteraction or intervention by a server. Such embodiments can beimplemented in one-device environments as well as in environments thatmay include more than one of either or both monitoring device and themonitored device. In one-device environments, the supervisor may log inwith different credentials so that he or she can view the inappropriatecontent that has been flagged and provide feedback to the analysis toolon the device. Where the monitoring and monitored device are separate,the inappropriate content as well as the derived data can be sentdirectly to the supervised device for further analysis, review andfeedback.

Various examples described herein are described in terms of a singlemonitoring device monitoring a single monitored device. However, inother embodiments, one or more monitoring devices can be used to monitorone or more monitored devices. For example, in some applications one ortwo managers may wish to monitor the performance of a large number ofemployees. As another example, either or both parents may want tomonitor the performance of one child. As yet another example, either orboth parents may want to use multiple devices as monitoring devices tomonitor multiple devices used by their child.

Where multiple devices are monitored by a single supervisor or group ofsupervisors, the same set of content tolerance levels (which may beimplemented as thresholds) can be distributed across a plurality ofmonitored devices. For example where a parent or parents are monitoringmultiple devices of a single child or multiple devices across multiplechildren, the parent or parents can decide to impose the same contenttolerance levels across all devices of all children. As another example,the parent or parents may set up one set of tolerance levels for thedevices or logins used by a younger child, and a different set oftolerance levels for the devices or logins used by an older child.Similarly, a manager may set up the same level of tolerance levelsacross all employees in his or her group, or may set up differenttolerance levels for different levels of employees in the group. Forexample, team leaders under the manager may have a different tolerancelevels as compared to workers under the team leaders.

As used herein, a circuit might be implemented utilizing any form ofhardware, software, or a combination thereof. For example, one or moreprocessors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logicalcomponents, software routines or other mechanisms might be implementedto make up a circuit. In implementation, the various circuits describedherein might be implemented as discrete circuits or the functions andfeatures described can be shared in part or in total among one or morecircuits. In other words, as would be apparent to one of ordinary skillin the art after reading this description, the various features andfunctionality described herein may be implemented in any givenapplication and can be implemented in one or more separate or sharedcircuits in various combinations and permutations. Even though variousfeatures or elements of functionality may be individually described orclaimed as separate circuits, one of ordinary skill in the art willunderstand that these features and functionality can be shared among oneor more common circuits, and such description shall not require or implythat separate circuits are required to implement such features orfunctionality.

Where circuits are implemented in whole or in part using software, inone embodiment, these software elements can be implemented to operatewith a computing or processing system capable of carrying out thefunctionality described with respect thereto. One such example computingsystem is shown in FIG. 14. Various embodiments are described in termsof this example-computing system 1400. After reading this description,it will become apparent to a person skilled in the relevant art how toimplement the technology using other computing systems or architectures.

Referring now to FIG. 14, computing system 1400 may represent, forexample, computing or processing capabilities found within desktop,laptop and notebook computers; hand-held computing devices (smartphones, cell phones, palmtops, tablets, etc.); mainframes,supercomputers, workstations or servers; or any other type ofspecial-purpose or general-purpose computing devices as may be desirableor appropriate for a given application or environment. Computing system1400 might also represent computing capabilities embedded within orotherwise available to a given device. For example, a computing systemmight be found in other electronic devices such as, for example, digitalcameras, navigation systems, cellular telephones, portable computingdevices, modems, routers, WAPs, terminals and other electronic devicesthat might include some form of processing capability.

Computing system 1400 might include, for example, one or moreprocessors, controllers, control modules, or other processing devices,such as a processor 1404. Processor 1404 might be implemented using ageneral-purpose or special-purpose processing engine such as, forexample, a microprocessor (whether single-, dual- or multi-coreprocessor), signal processor, graphics processor (e.g., GPU) controller,or other control logic. In the illustrated example, processor 1404 isconnected to a bus 1402, although any communication medium can be usedto facilitate interaction with other components of computing system 1400or to communicate externally.

Computing system 1400 might also include one or more memory modules,simply referred to herein as main memory 1408. For example, in someembodiments random access memory (RAM) or other dynamic memory, might beused for storing information and instructions to be executed byprocessor 1404. Main memory 1408 might also be used for storingtemporary variables or other intermediate information during executionof instructions to be executed by processor 1404. Computing system 1400might likewise include a read only memory (“ROM”) or other staticstorage device coupled to bus 1402 for storing static information andinstructions for processor 1404.

The computing system 1400 might also include one or more various formsof information storage mechanism such as storage devices 1410, whichmight include, for example, a media drive 1412 and a storage unitinterface 1420. The media drive 1412 might include a drive or othermechanism to support fixed or removable storage media 1414. For example,a hard disk drive, a floppy disk drive, a magnetic tape drive, anoptical disk drive, a CD or DVD drive (R or RW), a flash drive, or otherremovable or fixed media drive might be provided. Accordingly, storagemedia 1414 might include, for example, a hard disk, a floppy disk,magnetic tape, cartridge, optical disk, a CD or DVD, or other fixed orremovable medium that is read by, written to or accessed by media drive1412. As these examples illustrate, the storage media 1414 can include acomputer usable storage medium having stored therein computer softwareor data.

In alternative embodiments, information storage devices 1410 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing system 1400.Such instrumentalities might include, for example, a fixed or removablestorage unit 1422 and an interface 1420. Examples of such storage units1422 and interfaces 1420 can include a program cartridge and cartridgeinterface, a removable memory (for example, a flash memory or otherremovable memory module) and memory slot, a flash drive and associatedslot (for example, a USB drive), a PCMCIA slot and card, and other fixedor removable storage units 1422 and interfaces 1420 that allow softwareand data to be transferred from the storage unit 1422 to computingsystem 1400. Information storage devices 1410 may also include fixed orremovable RAM, ROM, or other memory.

Computing system 1400 might also include a communications interface1424. Communications interface 1424 might be used to allow software anddata to be transferred between computing system 1400 and externaldevices. Examples of communications interface 1424 might include a modemor softmodem, a network interface (such as an Ethernet, networkinterface card, WiMedia, IEEE 802.XX, Bluetooth® or other interface), acommunications port (such as for example, a USB port, IR port, RS232port, or other port), or other communications interface. Software anddata transferred via communications interface 1424 might typically becarried on signals, which can be electronic, electromagnetic (whichincludes optical) or other signals capable of being exchanged by a givencommunications interface 1424. These signals might be provided tocommunications interface 1424 via a channel 1428. This channel 1428might carry signals and might be implemented using a wired or wirelesscommunication medium. Some examples of a channel might include a phoneline, a cellular link, an RF link, an optical link, a network interface,a local or wide area network, and other wired or wireless communicationschannels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to media such as, forexample, memory 1408, storage unit 1420, media 1414, and channel 1428.These and other various forms of computer program media or computerusable media may be involved in carrying one or more sequences of one ormore instructions to a processing device for execution. Suchinstructions embodied on the medium, are generally referred to as“computer program code” or a “computer program product” (which may begrouped in the form of computer programs or other groupings). Whenexecuted, such instructions might enable the computing system 1400 toperform features or functions of the disclosed technology as discussedherein.

While various embodiments of the disclosed technology have beendescribed above, it should be understood that they have been presentedby way of example only, and not of limitation. Likewise, the variousdiagrams may depict an example architectural or other configuration forthe disclosed technology, which is done to aid in understanding thefeatures and functionality that can be included in the disclosedtechnology. The disclosed technology is not restricted to theillustrated example architectures or configurations, but the desiredfeatures can be implemented using a variety of alternative architecturesand configurations. Indeed, it will be apparent to one of skill in theart how alternative functional, logical or physical partitioning andconfigurations can be implemented to implement the desired features ofthe technology disclosed herein. Also, a multitude of differentconstituent module names other than those depicted herein can be appliedto the various partitions. Additionally, with regard to flow diagrams,operational descriptions and method claims, the order in which the stepsare presented herein shall not mandate that various embodiments beimplemented to perform the recited functionality in the same orderunless the context dictates otherwise.

Although the disclosed technology is described above in terms of variousexemplary embodiments and implementations, it should be understood thatthe various features, aspects and functionality described in one or moreof the individual embodiments are not limited in their applicability tothe particular embodiment with which they are described, but instead canbe applied, alone or in various combinations, to one or more of theother embodiments of the disclosed technology, whether or not suchembodiments are described and whether or not such features are presentedas being a part of a described embodiment. Thus, the breadth and scopeof the technology disclosed herein should not be limited by any of theabove-described exemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

What is claimed is:
 1. A system for device monitoring, comprising: anon-transitory machine-readable storage medium comprising instructionsexecutable by one or more processing units that, when executed, causethe system to: receive, from a monitored user device, derived datacorresponding to flagged media content detected by the monitored userdevice; analyze the derived data to determine if the flagged mediacontent corresponding to the derived data exceeds one or more mediacontent tolerance levels associated with the monitored device; andtransmit an alert to a second user device if the flagged media contentis determined to exceed one or more media content tolerance levels. 2.The system of claim 1, wherein monitored media content comprises ascreen capture of the monitored user device screen.
 3. The system ofclaim 1, wherein monitored media content comprises one or more of: audiooutput on the monitored user device, extracted text, and manuallyinputted text.
 4. The system of claim 1, wherein the instructions, whenexecuted by the one or more processing units, further cause the systemto receive media content from the monitored user device corresponding tothe flagged media content.
 5. The system of claim 4, wherein themonitored device encrypts the media content prior to transmitting it tothe server, such that the server is able to analyze data correspondingto the media content without accessing to the media content.
 6. Thesystem of claim 1, wherein the alert comprises flagged media contentdetermined to exceed a media content tolerance level, such that thesecond user device is able to view the flagged media content.
 7. Thesystem of claim 6, wherein the instructions, when executed by the one ormore processing units, further cause the system to receive, from thesecond user device, feedback comprising an indication as to whether theflagged media content constitutes inappropriate material according tostandards of the user.
 8. The system of claim 7, wherein the feedbackcomprises an indication as to whether the flagged media content shouldhave been flagged.
 9. The system of claim 7, wherein the feedbackcomprises an indication describing the flagged media content, whereinthe indication is received from the second user device and comprisingone or more of: a textual input describing characteristics of thecaptured media content; a rating of flagged media content; and a textualinput reciting text information contained in the captured media content.10. The system of claim 9, wherein the rating corresponds to one or moreof violence, sexual content, or language.
 11. A method for devicemonitoring, the method comprising the steps of: receiving, from amonitored user device, derived data corresponding to flagged mediacontent detected by the monitored user device; analyzing the deriveddata to determine if the flagged media content corresponding to thederived data exceeds one or more media content tolerance levelsassociated with the monitored device; and transmitting an alert to asecond user device if the flagged media content is determined to exceedone or more media content tolerance levels.
 12. The method of claim 11,wherein monitored media content comprises a screen capture of themonitored user device screen.
 13. The method of claim 11, whereinmonitored media content comprises one or more of: audio output on themonitored user device, extracted text, and manually inputted text. 14.The method of claim 11, further comprising receiving media content fromthe monitored user device corresponding to the flagged media content.15. The method of claim 14, wherein the monitored device encrypts themedia content prior to transmitting it to the server, such that theserver is able to analyze data corresponding to the media contentwithout accessing to the media content.
 16. The method of claim 11,wherein the alert comprises flagged media content determined to exceed amedia content tolerance level, such that the second user device is ableto view the flagged media content.
 17. The method of claim 16, furthercomprising receiving, from the second user device, feedback comprisingan indication as to whether the flagged media content constitutesinappropriate material according to standards of the user.
 18. Themethod of claim 17, wherein the feedback comprises an indication as towhether the flagged media content should have been flagged.
 19. Themethod of claim 17, wherein the feedback comprises an indicationdescribing the flagged media content, wherein the indication is receivedfrom the second user device and comprising one or more of: a textualinput describing characteristics of the captured media content; a ratingof flagged media content; a textual input reciting text informationcontained in the captured media content.
 20. The system of claim 19,wherein the rating corresponds to one or more of violence, sexualcontent, or language.